Brain tumor detector

Nutshell

As of now this project is under development.

In this project I build a program that detects and localizes cancer from images of human brains, as explained on the course Modern Artificial Intelligence, lectured by Dr. Ryan Ahmed, Ph.D. MBA.

The program will train two models which will

1. classify the images either containing cancer tumor or not
2. localizes the tumor within the brain

/content/drive/MyDrive/Colab Notebooks/brain-tumor-detector/Brain_MRI

Introduction to the Brain Tumor Detection

Deep learning has proven to be as good and even better than humans in detecting diseases from X-rays, MRI scans and CT scans. there is huge potential in using AI to speed up and improve the accuracy of diagnosis. This project will use the labeled dataset from https://www.kaggle.com/datasets/mateuszbuda/lgg-mri-segmentation which consists of 3929 Brain MRI scans and the tumor location. The final pipeline has a two step process where

1. A Resnet deep learning classifier model will classify the input images into two groups: tumor detected and tumor not detected.
2. For the images, where tumor was detected, a second step is performed, where a ResUNet segmentation model detects the tumor location on the pixel level.

Image segmentation

Image segmentation extracts information from images on the level of pixels. It is used for object recognition and localization in applications like medical imaging and self-driving cars. Image segmentation produces a pixel-wise mask of the image with deep learning approaches using common architectures such as CNN, FNNs and Deep Encoders-Decoders.

With Unet, the input and the output have the same size so the size of the images is preserved. In contrast to the CNN image classification, where the image is converted to a vector and the entire image is classified as a class label, the Unet performs classification on pixel level. Unet formulates a loss function for every pixel and then a softmax function is applied to every pixel. In other words, the segmentation problem is solved as a classification problem.

Looking into the data

We have a csv file that contains the patient IDs, the locations of the images, their masks and indicator if there is a tumor in the image (1 - tumor, 0 - healthy). There are 1373 images with tumors and 2556 healthy brain images. Thus, the dataset is imbalanced.

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 3929 entries, 0 to 3928
Data columns (total 4 columns):
 #   Column      Non-Null Count  Dtype 
---  ------      --------------  ----- 
 0   patient_id  3929 non-null   object
 1   image_path  3929 non-null   object
 2   mask_path   3929 non-null   object
 3   mask        3929 non-null   int64 
dtypes: int64(1), object(3)
memory usage: 122.9+ KB

	patient_id	image_path	mask_path	mask
0	TCGA_CS_5395_19981004	TCGA_CS_5395_19981004/TCGA_CS_5395_19981004_1.tif	TCGA_CS_5395_19981004/TCGA_CS_5395_19981004_1_...	0
1	TCGA_CS_5395_19981004	TCGA_CS_4944_20010208/TCGA_CS_4944_20010208_1.tif	TCGA_CS_4944_20010208/TCGA_CS_4944_20010208_1_...	0
2	TCGA_CS_5395_19981004	TCGA_CS_4941_19960909/TCGA_CS_4941_19960909_1.tif	TCGA_CS_4941_19960909/TCGA_CS_4941_19960909_1_...	0
3	TCGA_CS_5395_19981004	TCGA_CS_4943_20000902/TCGA_CS_4943_20000902_1.tif	TCGA_CS_4943_20000902/TCGA_CS_4943_20000902_1_...	0
4	TCGA_CS_5395_19981004	TCGA_CS_5396_20010302/TCGA_CS_5396_20010302_1.tif	TCGA_CS_5396_20010302/TCGA_CS_5396_20010302_1_...	0

brain_df.mask_path[1] # Path to the brain MRI image

'TCGA_CS_4944_20010208/TCGA_CS_4944_20010208_1_mask.tif'

brain_df.image_path[1] # Path to the segmentation mask

'TCGA_CS_4944_20010208/TCGA_CS_4944_20010208_1.tif'

	count
mask
0	2556
1	1373

dtype: int64

Visualisation of the datasets

Below is an exmaple of an MRI image and the matching mask. This example has a small tumor. In images where no tumor is present, the mask will be complety black.

Below are visualisations from 6 MRIs and their overlayed masks in rose color.

Convolutional neural networks (CNNs)

• The first CNN layers are used to extract high level general features
• The last couple of layers will perform classification
• Locla respective fields scan the image first searching for simple shapes such as edges and lines
• The edges are picked up by the subsequent layer to form more complex features Good visualisation of the feature extraction with convolutions can be found at https://setosa.io/ev/image-kernels/

ResNet (Residual Network)

• As CNNs grow deeper, vanishing gradients negatively imapct the network performance. Vanishing gradient occurs when the gradient is backpropagated to earlier layers which results in a very small gradient.
• ResNets "skip connection" feature can allow training of 152 layers wihtout vanishing gradient problems
• ResNet adds "identity mapping on top of the CNN
• ResNet deep network is trained with ImageNet, which contains 11 million images and 11 000 categories

ResNet paper (He etal, 2015): https://arxiv.org/pdf/1512.03385

As seen in the Figure 6. from the Resnet paper, the ResNet architectures overcome the training challenges from deep networks compared ot the plain networks. ResNet-152 achieved 3.58% error rate on the ImageNet dataset. This is better than human performance.

Siddarth Das has made agreat comparison of CNN architecture performances, you can check it out here: https://medium.com/analytics-vidhya/cnns-architectures-lenet-alexnet-vgg-googlenet-resnet-and-more-666091488df5

Transfer learning

Transfer learning retrains a network that has been trained to perform a specific task to use it in a similar task. The use of a pretrained model can drastically reduce the computational time and the amount of training data required, compared to starting from scratch. It can be compared to a salsa dancer starting to learn bachata; he/she will probably do a lot better than a person who has never danced before.

There are two main strategies in transfer learning:

1. Freeze the trained CNN network weights from the first layers and the train newly added dense layers. The new layers are initialized with random weights.
2. Retrain the entire CNN network while setting the learning rate to be very small. With too large learning rate the already trained weights might be changed too dramatically.

In this project I will use the approach 1.

Transfer learning has it's own challenges:

• Negative Transfer: the source task/domain is “close enough to look useful” but actually pushes the model in the wrong direction, hurting performance compared to training from scratch. This occurs when the features of old and new tasks are not related.
• Which layers to transfer / freeze: deciding what to reuse vs retrain is nontrivial; freezing too much can underfit, unfreezing too much can overfit or destabilize training.
• Representation misalignment: even if tasks are related, the internal features might not separate target classes well, especially when target cues differ (e.g., medical imaging vs natural images).
• Transfer bounds: Measuring the amount of knowledge transfered is crucial to ensure model quality and robustness. It is worth considering, how to quantify this, and it is a subject of ongoing research.

This is a great resource for transfer learning from Dipanjan Sarkar: https://towardsdatascience.com/a-comprehensive-hands-on-guide-to-transfer-learning-with-real-world-applications-in-deep-learning-212bf3b2f27a/

Part 1: Training a classifier model to detect if tumor exists or not

I use the flow_from_dataframe for training. Batch size = 16 class mode = categorical

# @title Preparing image generators
train_generator = datagen.flow_from_dataframe(
    dataframe = train,
    directory = './',
    x_col = 'image_path',
    y_col = 'mask',
    subset = 'training',
    batch_size =16,
    shuffle = True,
    class_mode = 'categorical',
    target_size = (256, 256)
)
valid_generator = datagen.flow_from_dataframe(
    dataframe = train,
    directory = './',
    x_col = 'image_path',
    y_col = 'mask',
    subset = 'validation',
    batch_size = 16,
    shuffle = True,
    class_mode = 'categorical',
    target_size = (256, 256)
)
#create a data generator for test images
#no need for splitting again because here we use the "test" data set
test_datagen = ImageDataGenerator(rescale=1./255)

test_generator = datagen.flow_from_dataframe(
    dataframe = test,
    directory = './',
    x_col = 'image_path',
    y_col = 'mask',
    batch_size = 16,
    shuffle = True,
    class_mode = 'categorical',
    target_size = (256, 256)
)

Found 2839 validated image filenames belonging to 2 classes.
Found 500 validated image filenames belonging to 2 classes.
Found 590 validated image filenames belonging to 2 classes.

# @title Retireve ResNet50 base model
#Input tensror 256 x 256 x 3
basemodel = ResNet50(weights = 'imagenet', include_top = False,
                     input_tensor = Input(shape = (256, 256, 3)))

Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/resnet/resnet50_weights_tf_dim_ordering_tf_kernels_notop.h5
94765736/94765736 ━━━━━━━━━━━━━━━━━━━━ 0s 0us/step

Below is the architecture of the ResNet50 model. For the transfer learning, all of these layers will be set to trainable = False to stop the weights from changing.

# Add classification head to the base model
headmodel = basemodel.output
headmodel = AveragePooling2D(pool_size = (4, 4))(headmodel)
headmodel = Flatten(name = 'flatten')(headmodel)
headmodel = Dense(256, activation = 'relu')(headmodel)
headmodel = Dropout(0.3)(headmodel)
headmodel = Dense(2, activation = 'softmax')(headmodel)

fullmodel = Model(inputs = basemodel.input, outputs = headmodel)

# compile the model
fullmodel.compile(loss = 'categorical_crossentropy', optimizer='adam',
              metrics=["accuracy"])

# use the early stopping to exit training
earlystopping = EarlyStopping(monitor='val_loss', mode='min', verbose = 1,
                              patience = 20)
# save the best model with least validation loss
checkpointer = ModelCheckpoint(filepath='classifier-resnet-weights.keras',
                               verbose=1, save_best_only=True)

history = fullmodel.fit(train_generator,
            steps_per_epoch = train_generator.n // train_generator.batch_size,
            epochs=1,
            validation_data=valid_generator,
            validation_steps= valid_generator.n // valid_generator.batch_size,
            callbacks=[checkpointer, earlystopping])

/usr/local/lib/python3.12/dist-packages/keras/src/trainers/data_adapters/py_dataset_adapter.py:121: UserWarning:

Your `PyDataset` class should call `super().__init__(**kwargs)` in its constructor. `**kwargs` can include `workers`, `use_multiprocessing`, `max_queue_size`. Do not pass these arguments to `fit()`, as they will be ignored.

177/177 ━━━━━━━━━━━━━━━━━━━━ 0s 18s/step - accuracy: 0.6947 - loss: 1.1854 
Epoch 1: val_loss improved from inf to 0.68954, saving model to classifier-resnet-weights.keras
177/177 ━━━━━━━━━━━━━━━━━━━━ 3484s 19s/step - accuracy: 0.6949 - loss: 1.1833 - val_accuracy: 0.6552 - val_loss: 0.6895